{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# **Rewrite-Retrieve-Read (RRR)**\n",
        "Rewrite-Retrieve-Read is a three-step framework for tasks that involve retrieval augmentation, such as open-domain question answering. It focuses on improving the quality of retrieved information and generating accurate outputs by refining the input query.\n",
        "\n",
        "Research Paper: [Rewrite-Retrieve-Read](https://arxiv.org/pdf/2305.14283)"
      ],
      "metadata": {
        "id": "SO0vrYMzBAEt"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Initial Setup**"
      ],
      "metadata": {
        "id": "C9ppe73RTGSc"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "! pip install --q athina chromadb"
      ],
      "metadata": {
        "id": "7HE6-T-8TFQo"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# ! pip install --q athina datasets langchain_community  langchain-openai langchainhub chromadb langchain"
      ],
      "metadata": {
        "id": "tN9wwP2cCFmW"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "from google.colab import userdata\n",
        "os.environ[\"OPENAI_API_KEY\"] = userdata.get('OPENAI_API_KEY')\n",
        "os.environ['ATHINA_API_KEY'] = userdata.get('ATHINA_API_KEY')"
      ],
      "metadata": {
        "id": "HD6QjehgA_P5"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Indexing**"
      ],
      "metadata": {
        "id": "tP1ZFteNTN7K"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TS-Th7YUAVWG"
      },
      "outputs": [],
      "source": [
        "# load embedding model\n",
        "from langchain_openai import OpenAIEmbeddings\n",
        "embeddings = OpenAIEmbeddings()"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# load data\n",
        "from langchain.document_loaders import CSVLoader\n",
        "loader = CSVLoader(\"./context.csv\")\n",
        "documents = loader.load()"
      ],
      "metadata": {
        "id": "EqCSPjoLChwv"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# split documents\n",
        "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
        "text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=0)\n",
        "documents = text_splitter.split_documents(documents)"
      ],
      "metadata": {
        "id": "sPS52vUdCjqt"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# create vectorstore\n",
        "from langchain.vectorstores import Chroma\n",
        "vectorstore = Chroma.from_documents(documents, embeddings)"
      ],
      "metadata": {
        "id": "3h0Ui6uGClnW"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Retriever**"
      ],
      "metadata": {
        "id": "0U0xQG8jT2l9"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# create retriever\n",
        "retriever = vectorstore.as_retriever()"
      ],
      "metadata": {
        "id": "L20M2_CCCn49"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **RAG Chain**"
      ],
      "metadata": {
        "id": "5OyGAQhTT9UK"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# load llm\n",
        "from langchain_openai import ChatOpenAI\n",
        "llm = ChatOpenAI()"
      ],
      "metadata": {
        "id": "bo5gAxDADFmn"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# create document chain\n",
        "from langchain.prompts import ChatPromptTemplate\n",
        "from langchain.schema.runnable import RunnablePassthrough\n",
        "from langchain.schema.output_parser import StrOutputParser\n",
        "\n",
        "template = \"\"\"\"\n",
        "You are a helpful assistant that answers questions based on the following context.\n",
        "If you don't find the answer in the context, just say that you don't know.\n",
        "Context: {context}\n",
        "\n",
        "Question: {input}\n",
        "\n",
        "Answer:\n",
        "\n",
        "\"\"\"\n",
        "prompt = ChatPromptTemplate.from_template(template)\n",
        "\n",
        "\n",
        "rag_chain = (\n",
        "    {\"context\": retriever,  \"input\": RunnablePassthrough()}\n",
        "    | prompt\n",
        "    | llm\n",
        "    | StrOutputParser()\n",
        ")"
      ],
      "metadata": {
        "id": "fXtQRxFSDNec"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Simple Query**"
      ],
      "metadata": {
        "id": "xRmlBMZFVg5H"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# define simple query\n",
        "simple_query = \"who directed the matrix\""
      ],
      "metadata": {
        "id": "T3Bst4XBDlZ8"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# response\n",
        "response = rag_chain.invoke(simple_query)\n",
        "response"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "0o_gT5PPDgql",
        "outputId": "2f17dbfa-7a84-4379-d55f-c6357170ec17"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'The Matrix was directed by the Wachowskis.'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Distracted Query**"
      ],
      "metadata": {
        "id": "V3JS_nBmVmwT"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# define distracted query\n",
        "distracted_query = \"who create the matrics\""
      ],
      "metadata": {
        "id": "SIwphWquDv9E"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# response\n",
        "response = rag_chain.invoke(distracted_query)\n",
        "response"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "p-2Rq06TEALm",
        "outputId": "59bc1a76-edc1-4262-a717-0fd7ff64e6fd"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "\"I don't know.\""
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Rewrite Retrieve Read**"
      ],
      "metadata": {
        "id": "e-4zJU8LVrJz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# define rewrite prompt for distracted query\n",
        "\n",
        "template = \"\"\"Provide a better search query for \\\n",
        "web search engine to answer the given question, end \\\n",
        "the queries with ’**’. Question: \\\n",
        "{x} Answer:\"\"\"\n",
        "\n",
        "rewrite_prompt = ChatPromptTemplate.from_template(template)"
      ],
      "metadata": {
        "id": "82XnIBLQE3r_"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# parse response\n",
        "def _parse(text):\n",
        "    return text.strip('\"').strip(\"**\")"
      ],
      "metadata": {
        "id": "eE8VflngFcRu"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# create rewriter chain\n",
        "rewriter = rewrite_prompt | ChatOpenAI(temperature=0) | StrOutputParser() | _parse"
      ],
      "metadata": {
        "id": "_Cg_zYy5FAkB"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# updated query\n",
        "rewriter.invoke({\"x\": distracted_query})"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "KOFlwsq3FKWT",
        "outputId": "026bcbe4-9219-4c76-c870-3031cb95e1c2"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'Who is the creator of the Matrix film series?'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# create rewrite retrieve read chain\n",
        "rewrite_retrieve_read_chain = (\n",
        "    {\n",
        "        \"context\": {\"x\": RunnablePassthrough()} | rewriter | retriever,\n",
        "        \"input\": RunnablePassthrough(),\n",
        "    }\n",
        "    | prompt\n",
        "    | llm\n",
        "    | StrOutputParser()\n",
        ")"
      ],
      "metadata": {
        "id": "DsfoEcsJGBsJ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# final response\n",
        "rewrite_retrieve_read_chain.invoke(distracted_query)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "NMDFiHBKFlsz",
        "outputId": "20b89c91-96b0-4a3b-92b2-ce0c32b0489e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'The Matrix was created by the Wachowskis.'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Preparing Data for Evaluation**"
      ],
      "metadata": {
        "id": "-H_IJfdsYIRv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# create data\n",
        "response = []\n",
        "contexts = []\n",
        "questions = []\n",
        "\n",
        "rewritten_query = rewriter.invoke({\"x\": distracted_query})\n",
        "questions.append(rewritten_query)\n",
        "response.append(rewrite_retrieve_read_chain.invoke(distracted_query))\n",
        "contexts.append([docs.page_content for docs in retriever.get_relevant_documents(rewritten_query)])\n",
        "\n",
        "\n",
        "data = {\n",
        "    \"query\": questions,\n",
        "    \"response\": response,\n",
        "    \"context\": contexts,\n",
        "}"
      ],
      "metadata": {
        "id": "RdR9KRSCWm5T"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# create dataset\n",
        "from datasets import Dataset\n",
        "dataset = Dataset.from_dict(data)"
      ],
      "metadata": {
        "id": "gBS2PKAgXXWc"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# create dataframe\n",
        "import pandas as pd\n",
        "df = pd.DataFrame(dataset)"
      ],
      "metadata": {
        "id": "fEAxNCj4XdgW"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 132
        },
        "id": "gNyS_qn-XeME",
        "outputId": "476fc3ff-bc7d-4457-f0b6-c54c450b8ae9"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                           query  \\\n",
              "0  Who is the creator of the Matrix film series?   \n",
              "\n",
              "                                    response  \\\n",
              "0  The Matrix was created by the Wachowskis.   \n",
              "\n",
              "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               context  \n",
              "0  ['The Matrix is a 1999 science fiction action film written and directed by the Wachowskis. It is the first installment in the Matrix film series, starring Keanu Reeves, Laurence Fishburne, Carrie-Anne Moss, Hugo Weaving, and Joe Pantoliano, and depicts a dystopian future in which humanity is unknowingly trapped inside the Matrix, a simulated reality that intelligent machines have created to distract humans while using their bodies as an energy source. When computer programmer Thomas Anderson...  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-0efc29bf-e489-4698-80ed-e01d4decbd11\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>query</th>\n",
              "      <th>response</th>\n",
              "      <th>context</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Who is the creator of the Matrix film series?</td>\n",
              "      <td>The Matrix was created by the Wachowskis.</td>\n",
              "      <td>['The Matrix is a 1999 science fiction action film written and directed by the Wachowskis. It is the first installment in the Matrix film series, starring Keanu Reeves, Laurence Fishburne, Carrie-Anne Moss, Hugo Weaving, and Joe Pantoliano, and depicts a dystopian future in which humanity is unknowingly trapped inside the Matrix, a simulated reality that intelligent machines have created to distract humans while using their bodies as an energy source. When computer programmer Thomas Anderson...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-0efc29bf-e489-4698-80ed-e01d4decbd11')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-0efc29bf-e489-4698-80ed-e01d4decbd11 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-0efc29bf-e489-4698-80ed-e01d4decbd11');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "  <div id=\"id_f0dd5a1f-cf74-4d85-8a36-de24371a11a5\">\n",
              "    <style>\n",
              "      .colab-df-generate {\n",
              "        background-color: #E8F0FE;\n",
              "        border: none;\n",
              "        border-radius: 50%;\n",
              "        cursor: pointer;\n",
              "        display: none;\n",
              "        fill: #1967D2;\n",
              "        height: 32px;\n",
              "        padding: 0 0 0 0;\n",
              "        width: 32px;\n",
              "      }\n",
              "\n",
              "      .colab-df-generate:hover {\n",
              "        background-color: #E2EBFA;\n",
              "        box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "        fill: #174EA6;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate {\n",
              "        background-color: #3B4455;\n",
              "        fill: #D2E3FC;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate:hover {\n",
              "        background-color: #434B5C;\n",
              "        box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "        filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "        fill: #FFFFFF;\n",
              "      }\n",
              "    </style>\n",
              "    <button class=\"colab-df-generate\" onclick=\"generateWithVariable('df')\"\n",
              "            title=\"Generate code using this dataframe.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M7,19H8.4L18.45,9,17,7.55,7,17.6ZM5,21V16.75L18.45,3.32a2,2,0,0,1,2.83,0l1.4,1.43a1.91,1.91,0,0,1,.58,1.4,1.91,1.91,0,0,1-.58,1.4L9.25,21ZM18.45,9,17,7.55Zm-12,3A5.31,5.31,0,0,0,4.9,8.1,5.31,5.31,0,0,0,1,6.5,5.31,5.31,0,0,0,4.9,4.9,5.31,5.31,0,0,0,6.5,1,5.31,5.31,0,0,0,8.1,4.9,5.31,5.31,0,0,0,12,6.5,5.46,5.46,0,0,0,6.5,12Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "    <script>\n",
              "      (() => {\n",
              "      const buttonEl =\n",
              "        document.querySelector('#id_f0dd5a1f-cf74-4d85-8a36-de24371a11a5 button.colab-df-generate');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      buttonEl.onclick = () => {\n",
              "        google.colab.notebook.generateWithVariable('df');\n",
              "      }\n",
              "      })();\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "df",
              "summary": "{\n  \"name\": \"df\",\n  \"rows\": 1,\n  \"fields\": [\n    {\n      \"column\": \"query\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"Who is the creator of the Matrix film series?\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"response\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"The Matrix was created by the Wachowskis.\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"context\",\n      \"properties\": {\n        \"dtype\": \"object\",\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 38
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Convert to dictionary\n",
        "df_dict = df.to_dict(orient='records')\n",
        "\n",
        "# Convert context to list\n",
        "for record in df_dict:\n",
        "    if not isinstance(record.get('context'), list):\n",
        "        if record.get('context') is None:\n",
        "            record['context'] = []\n",
        "        else:\n",
        "            record['context'] = [record['context']]"
      ],
      "metadata": {
        "id": "26r0OY_YYGfw"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Evaluation in Athina AI**\n",
        "\n",
        "We will use **Answer Relevancy** eval here. It Measures how pertinent the generated response is to the given prompt. Please refer to our [documentation](https://docs.athina.ai/api-reference/evals/preset-evals/overview) for further details"
      ],
      "metadata": {
        "id": "HaBz2noKZP7C"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# set api keys for Athina evals\n",
        "from athina.keys import AthinaApiKey, OpenAiApiKey\n",
        "OpenAiApiKey.set_key(os.getenv('OPENAI_API_KEY'))\n",
        "AthinaApiKey.set_key(os.getenv('ATHINA_API_KEY'))"
      ],
      "metadata": {
        "id": "s0nAgnCtYw_s"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# load dataset\n",
        "from athina.loaders import Loader\n",
        "dataset = Loader().load_dict(df_dict)"
      ],
      "metadata": {
        "id": "XTnWXK9_ZlR6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# evaluate\n",
        "from athina.evals import RagasAnswerRelevancy\n",
        "RagasAnswerRelevancy(model=\"gpt-4o\").run_batch(data=dataset).to_df()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 393
        },
        "id": "7l5rGh5LZnA8",
        "outputId": "d1310618-80c4-4437-86e2-a4326010b46b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "evaluating with [answer_relevancy]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "100%|██████████| 1/1 [00:01<00:00,  1.08s/it]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "You can view your dataset at: https://app.athina.ai/develop/ddec3010-12e6-4f5e-bbc0-188934bc90dc\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                           query  \\\n",
              "0  Who is the creator of the Matrix film series?   \n",
              "\n",
              "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               context  \\\n",
              "0  ['The Matrix is a 1999 science fiction action film written and directed by the Wachowskis. It is the first installment in the Matrix film series, starring Keanu Reeves, Laurence Fishburne, Carrie-Anne Moss, Hugo Weaving, and Joe Pantoliano, and depicts a dystopian future in which humanity is unknowingly trapped inside the Matrix, a simulated reality that intelligent machines have created to distract humans while using their bodies as an energy source. When computer programmer Thomas Anderson...   \n",
              "\n",
              "                                    response expected_response  \\\n",
              "0  The Matrix was created by the Wachowskis.              None   \n",
              "\n",
              "             display_name failed  \\\n",
              "0  Ragas Answer Relevancy   None   \n",
              "\n",
              "                                                                                                                                                                                                                                                                  grade_reason  \\\n",
              "0  A response is deemed relevant when it directly and appropriately addresses the original query. Importantly, our assessment of answer relevance does not consider factuality but instead penalizes cases where the response lacks completeness or contains redundant details   \n",
              "\n",
              "   runtime   model  ragas_answer_relevancy  \n",
              "0     1493  gpt-4o                0.961197  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-56545a70-02e0-464b-9ffb-278e18068546\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>query</th>\n",
              "      <th>context</th>\n",
              "      <th>response</th>\n",
              "      <th>expected_response</th>\n",
              "      <th>display_name</th>\n",
              "      <th>failed</th>\n",
              "      <th>grade_reason</th>\n",
              "      <th>runtime</th>\n",
              "      <th>model</th>\n",
              "      <th>ragas_answer_relevancy</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Who is the creator of the Matrix film series?</td>\n",
              "      <td>['The Matrix is a 1999 science fiction action film written and directed by the Wachowskis. It is the first installment in the Matrix film series, starring Keanu Reeves, Laurence Fishburne, Carrie-Anne Moss, Hugo Weaving, and Joe Pantoliano, and depicts a dystopian future in which humanity is unknowingly trapped inside the Matrix, a simulated reality that intelligent machines have created to distract humans while using their bodies as an energy source. When computer programmer Thomas Anderson...</td>\n",
              "      <td>The Matrix was created by the Wachowskis.</td>\n",
              "      <td>None</td>\n",
              "      <td>Ragas Answer Relevancy</td>\n",
              "      <td>None</td>\n",
              "      <td>A response is deemed relevant when it directly and appropriately addresses the original query. Importantly, our assessment of answer relevance does not consider factuality but instead penalizes cases where the response lacks completeness or contains redundant details</td>\n",
              "      <td>1493</td>\n",
              "      <td>gpt-4o</td>\n",
              "      <td>0.961197</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-56545a70-02e0-464b-9ffb-278e18068546')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-56545a70-02e0-464b-9ffb-278e18068546 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-56545a70-02e0-464b-9ffb-278e18068546');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "repr_error": "Out of range float values are not JSON compliant: nan"
            }
          },
          "metadata": {},
          "execution_count": 41
        }
      ]
    }
  ]
}